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ABSTRACT 

Continuous, formative evaluation, from inception to completion, with 
rapid revisions, was key to the successful development of Kids Design Network (KDN) by 
the DuPage Children's Museum (DCM) . Although a thorough evaluation plan was developed 
at the beginning of the project, it soon became apparent that additional strategies 
were necessary to insure success. As new issues were identified and planned techniques 
were found inadequate, new evaluation techniques were developed and implemented. The 
development of KDN illustrates the importance of flexible, formative evaluation from 
the inception of a program to its completion. KDN is an interactive Internet-based 
learning program for elementary aged students. Using the KDN Web site, students work 
collaboratively with an engineer {via real-time communication on the KDN Engineer 
Chatboard) , their classmates and teacher to design, build and test a gadget that meets 
an engineering challenge. Interactive portions of the Web site include a drawing 
program where students draw a design for their gadget and a whiteboard with chat where 
children and engineers can mark on the student's design and communicate via text chat. 
DCM brought in an evaluation consultant in the very early stages of KDN development. 
The evaluator became an integral part of development and continuously collaborated 
with the team. The collaborative nature of the evaluation and the fluidity of the plan 
ultimately produced a workable, user friendly Web site consistent with the museum's 
mission. (Author) 
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Continuous, formative evaluation, from inception to completion, with rapid 
revisions, was key to the successful development of Kids Design Network 
(KDN) by the DuPage Children's Museum (DCM). Although a thorough 
evaluation plan was developed at the beginning of the project, it soon became 
apparent that additional strategies were necessary to insure success. As new 
issues were identified and planned techniques were found inadequate, new 
evaluation techniques were developed and implemented. The development of 
KDN illustrates the importance of flexible, formative evaluation from the 
inception of a program to its completion. 



KDN is an interactive Internet-based learning program for elementary aged 
students. Using the KDN Web site 

(htt p://www .dupagechildrensmuseum.org /kdn ). students work collaboratively 
with an engineer (via real-time communication on the KDN Engineer 
Chatboard), their classmates and teacher to design, build and test a gadget 
that meets an engineering challenge. Interactive portions of the Web site 
include a drawing program where students draw a design for their gadget and a 
whiteboard with chat where children and engineers can mark on the student's 
design and communicate via text chat. 
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DCM brought in an evaluation consultant in the very early stages of KDN 
development. The evaluator became an integral part of development and 
continuously collaborated with the team. The collaborative nature of the 
evaluation and the fluidity of the plan ultimately produced a workable, user 
friendly Web site consistent with the museum's mission. 
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The Internet's potential for linking students to experts in a field has long been 
recognized. Many museums and government agencies (including the DuPage 
Children's Museum) have outreach programs based on a Web site where information 
is presented by the institution and/or the students, and students and experts 
communicate via e-mail. Kids Design Network (KDN) takes the important next step 
and makes this two-way communication live, secure , and integral to the learning 
process. The DuPage Children's Museum (DCM) conceived KDN as a way to 
encourage and facilitate the incorporation of design technology, open-ended problem 
solving, and engineering and computer skills into elementary school classrooms. 
Basing KDN on the Web extended the Museum's reach well beyond the Chicago 
area. 



Working with young children as DCM does has shown the importance of continuous 
evaluation of new programs in scenarios as closely approximating actual use as 
possible. When one combines the unpredictability of children with the wide variety 
and constantly changing nature of classroom settings and computer technology, 
flexibility in this continuous evaluation becomes of paramount importance. 
Continuous, formative evaluation with rapid revisions from inception to completion 
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was key to the successful development of KDN. 

The basis for KDN is DCM's Kids Design Engineering (KDE) Learning Lab in which 
students are challenged to design, build and test an air (balloon) powered vehicle. 
Now in its seventh year, KDE has been thoroughly tested and is highly successful at 
teaching elementary students design engineering and problem solving skills. KDE is 
heavily facilitated by DCM personnel who visit more than 60 classrooms each year, 
including those in the local school districts where KDE is incorporated into the 3 rd 
grade curriculum. The plan for KDN was to take the KDE model and substitute an on- 
line engineer for the DCM facilitators. Using the Internet, students could communicate 
in real-time with the engineer, asking him/her questions about the design of their 
projects. Based on this idea, DCM applied for and received KDN pilot funding from 
the Illinois State Board of Education Center for Scientific Literacy. 



Program Description 
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Figure 1: How KDN works . 

Students can consult with the on-line engineer at any time during the design and 
building process. Many teachers have their students talk to the engineer more than 
once. The real-time communication provides students with usable information as they 
design and build their gadgets. 

.On the KDN Web site (http://www.dupagechildrensmuseum.org/kdn), engineers 
introduce a challenge and relate it to a real-life engineering challenge they have 
faced. Table 1 summarizes current challenges. Elementary aged students work 
collaboratively with an engineer (via real-time communication on the KDN Engineer 
Chatboard), their classmates and teacher to design a gadget that meets the 
challenge. Students use KDN's on-line drawing program to draw a design for their 
gadget. On the KDN Engineer Chatboard, both students and engineer see the 
design, mark on it, and discuss it via live text chat. Students build their gadget using 
materials found in their recycle bins, then test and modify it with help from the on-line 
engineer, teacher and classmates. 



Challenge Task 

Puppet Build a puppet that moves in at least 2 ways 

Chemical engineering a g ac jg e t to sort 3 different types of objects, such as 
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paper clips, marbles and crayons 


Slowly Moving 
Marble 


Build a pathway for a marble to roll down as slowly as possible 


Nuclear Engineering 


Build a gadget to retrieve something from about 3 feet away 
from you 


Door Alarm 


Build an alarm for your bedroom door 


Civil Engineering 


Build a gadget that allows you to pour water from a jug without 
touching the jug 



Table 1 . Engineering challenges currently posted on the KDN Web site . 
Evaluation philosophy of DCM 

All programs and exhibits are continuously evaluated at DCM. Revisions are made 
based on the evaluation, and then are themselves evaluated. No matter how small 
the project or how tight the budget, evaluation starts at the first brainstorming session 
and never really ends. In developing both KDN itself and its evaluation plan, we found 
that we were following the engineering method that KDN is designed to teach: design, 
build, test, redesign, build, test again, redesign... . In a project as complex as KDN 
turned out to be, ongoing evaluation of both macro and micro components was 
imperative. 

This philosophy continues to permeate the development of KDN, and typically 
includes professionals from many different backgrounds. Groups involved in 
evaluating KDN included: 

• In-house members of the KDN team and other DCM employees; 

• An Advisory Board drawn from the community and consisting of engineers; 
computer experts; experts in Web-based school-university-partnerships; and 
school district administrators, principals, teachers and curriculum 
coordinators; 

• Students and teachers using KDN; 

• Web designers and programmers contracted to produce the Web site; 

• Other professional Web designers; 

• An independent, professional evaluator. 

The KDN team consisted of the DCM project manager and experienced KDE 
facilitators (including engineers, scientists, and former classroom teachers), the 
independent evaluator, and the contracted Web site developer’s programmer and 
project director. This group worked closely together for 2 years to develop KDN. 

DCM Employees 

In-house members of the KDN team included former classroom teachers, engineers, 
scientists, and exhibit and program designers. They were experienced in conducting 
effective science programming for elementary children, as well as expert in the 
development of open-ended problem-solving activities. Other DCM employees 
involved included facilitators of school and museum floor programs, art specialists, 
exhibit developers, and DCM management. They provided fresh eyes to review all 
aspects of KDN, and were a ready source of Web site testers. Since these people 
were not involved in the day-to-day development, they provided a different 
perspective on everything from Web site graphics to navigation to teacher guidance 
to ensure KDN was consistent with DCM’s mission. 

Advisory Board 

DCM regularly uses community-based advisory boards when developing new 
programs and exhibits. KDN’s advisory board was especially important in providing 
philosophical and practical guidance in the early stages of development. School 
officials on the advisory board helped insure that KDN would be useful to and usable 
by classroom teachers. Principals and curriculum coordinators helped pilot test KDN 
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in their school's classrooms. Computer experts gave advice on available technology. 
Experts in school-university-professional partnerships provided information on 
existing collaborative networks, how these collaborations worked, and potential 
pitfalls KDN might face. Engineers helped provide Challenges and advice on how to 
staff the on-line engineer positions. Since engineers would be using the Web site, 
their input on the components necessary for effective communication were 
invaluable. 

Students and Teachers 

it is part of the culture of DCM that all programs and exhibits are pilot tested first in 
small groups and then in the setting w successful development. Because teachers 
are the primary users of KDN, a strong emphasis was placed on meeting their needs. 
Our ehere they will be used. Feedback from the users is absolutely necessary 
tovaluation methods needed to identify teacher concerns and ways to alleviate them. 
We found that teachers’ opinions/reactions to KDN after using it with children were 
markedly different from their reactions before using it. For example, teachers greatly 
underestimated students’ enthusiasm for talking with the engineer and the value of 
the building process. We therefore tested KDN in classrooms and computer labs with 
entire classes of children. We held preliminary briefing sessions with teachers, 
observed and talked with them while their students were using KDN, and whenever 
possible, talked with them afterwards. Ultimately, 40 classroom teachers participated 
in the development of KDN. Their input was a critical factor in development. 

We knew in advance that elementary teachers have very little discretionary time and 
are under constant pressure to demonstrate that their students perform well on 
standardized tests. We needed to ensure that teachers could see the benefits of KDN 
in achieving educational goals. We also needed to ensure teachers were given 
enough information to feel comfortable using KDN on their own. Many teachers are 
not accustomed to using hands-on activities in their classrooms, and many are not 
adept at using computers. 

Pilot testing with children is an absolute necessity because of their widely varying skill 
levels, interests, and experiences. Sometimes our hypotheses were confirmed in the 
tests, but frequently we were surprised. Testing with large numbers of children is 
critical for a program intended for use in classrooms. Children function differently in 
large classroom settings than they do in small groups in or out of school. Tests in 
schools also gave important information on Web site performance on school 
computers with many children (up to 30) using the site’s interactive features at once. 
Every part of KDN was extensively child tested, with over 900 children eventually 
involved. 

Web Site Developers and Programmers 

When KDN development began, DCM had no in-house Web site development 
expertise. Many potential contractors were considered, with emphasis on identifying 
one which could provide interactive, innovative and engaging programming for 
children. Educational Web Adventures, LLP (Eduweb) was chosen. Eduweb worked 
closely with other members of the KDN team in developing the evaluation plan as 
well as the Web site. In addition to normal extensive preliminary testing of Web site 
components, Eduweb’s programmer and project director were in the classroom 
observing children for the first full-scale test of the KDN. Subsequently, they 
frequently served as on-line engineers, getting first hand understanding of how the 
Engineer chatboard works. The programmer continues to periodically monitor 
engineer chat sessions and trouble-shoot as needed. 

Independent Evaluator 

When given the choice, many museums conduct front-end and formative evaluation 
in-house and save their budgets to bring in an independent evaluator at the 
summative stage when the project is complete. DCM brought in an independent 
evaluator at the very early stages of KDN development. This designated one person 
as the focus for evaluation, and the evaluator became an integral part of the KDN 
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team. She helped define the evaluation methods used, interpreted the data, and 
provided detailed interim and final reports needed to satisfy grant requirements. 
Involvement from the beginning of the project gave her valuable longitudinal 
perspective on development. She offered many suggestions that were implemented 
as KDN progressed. Had she been brought in at the conclusion of the project (after 
the grant money was spent), DCM would not have been as able to take advantage of 
her input. 

Working with the KDN project manager, the independent evaluator developed a 
thorough evaluation plan very early in the KDN development process. The plan 
detailed her role in the process, the activities she would conduct, the data collection 
methods to be used, and a topical framework outlining the questions her evaluation 
would address. The topical framework was used to focus and guide the evaluator’s 
observations and conversations with respondents, but as important new issues 
emerged, these were often explored as well. 

The independent evaluator and DCM staff employed a naturalistic methodology to 
collect and analyze data (Lincoln and Guba, 1985). Naturalistic inquiry is particularly 
useful when seeking to understand real life settings and the situations and ways in 
which people experience them. It begins with a focus of study, and thorough 
systematic collection and analysis of the data identifies and reports on issues as they 
emerge. Data is collected from a variety of sources. This ’’triangulation ” of data yields 
the completest possible description of the setting and program. Naturalistic inquiry 
often includes the respondents’ own language as a way of painting a rich, detailed, 
and meaningful picture of their experience. 

Ultimately, evaluation became a very collaborative and continuous effort. Although 
the independent evaluator was not present at all tests, she was present for most of 
the tests where major revisions to the program were assessed. She was also present 
at the meetings where the team took stock of the program’s status and charted the 
course for the next stage of development and evaluation. The project manager kept 
her constantly informed via e-mail and phone. Information and opinions freely flowed 
to and from the evaluator, project manager, and Web designers. 

The Evaluation Process 

Although DCM already had an e-mail based math program on the Web and was 
experienced in conducting engineering programs for children, KDN was a venture into 
uncharted territory. Because there were so many aspects of the program to evaluate, 
it was helpful to isolate components and design evaluation to answer specific 
questions. Most components were assessed individually first and then incorporated 
into a trial of the whole KDN experience. For example, when the Slowly Moving 
Marble Challenge was being developed, DCM employees’ children in the museum on 
a school holiday were the first to try their hands at designing and building a pathway 
to slow down a rolling marble. These children didn’t use the computer; they simply 
built their pathways. A few months later, a 3 rd -grade school class (taught by an KDN 
Advisory Board member) took on the marble challenge as part of the total KDN 
experience. Web site components were also evaluated in isolation first. Numerous 
versions of the Engineer chatboard were tested by the Web designers, then DCM 
staff, before children ever encountered them. The first children to see the chatboard 
were the project manager’s children visiting the museum. Many iterations later, the 
first classroom trials of the chatboard occurred. 

At the outset, three overriding questions were identified. Evaluation would be geared 
toward answering these questions: 

1 . Is KDN technologically feasible? 

2. Would teachers and kids be interested in talking to the engineer? 

3. What, exactly, should the content of the Web site be? 

These questions and how we answered them are discussed below. Methods are 
summarized in Table 2. 
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Challenge 


Task 


Puppet 


Build a puppet that moves in at least 2 ways 


Chemical 

engineering 


Build a gadget to sort 3 different types of objects, such as 
paper clips, marbles and crayons 


Slowly Moving 
Marble 


Build a pathway for a marble to roll down as slowly as 
possible 


Nuclear Engineering 


Build a gadget to retrieve something from about 3 feet away 
from you 


Door Alarm 


Build an alarm for your bedroom door 


Civil Engineering 


Build a gadget that allows you to pour water from a jug without 
touching the jug 



Table 2: Evaluation methods used during KDN development 
Is KDN technologically feasible? 



The first question to be answered was whether live two-way communication was even 
technically possible in the school setting. Three primary concerns were quickly 
identified: 

1 . The cognitive development of young children and their resultant ability to 
communicate would be key factors in determining the appropriate age to use 
KDN. KDN was initially aimed at children in 1 st through 5 th grades. The 
younger children in particular typically have poor typing skills, and have 
trouble communicating with someone not in the same room with them. The 
appropriate age for children to use KDN became one of the most significant 
questions to be answered. 

2. Elementary schools typically do not have the latest in computer technology. It 
is not uncommon for computers to be more than 5 years old. Would KDN 
work on Windows 95 and Internet Explorer 3.5? At the time of development, 
many schools didn’t have Flash technology. What could be used to make the 
Web site look appealing to kids and teachers? A key obstacle was firewalls in 
schools that block the streaming data necessary for the two-way 
communication at the heart of KDN. Would it be possible to tunnel through the 
firewalls, or would KDN use be restricted to school districts without them? 

3. If the first two issues could be resolved, the safety of the children and 
consequent security of the communication became major issues. The 
Museum and KDN users needed assurance that children would be protected 
from inappropriate communications and unauthorized users. 



Would kids and teachers be interested in talking to the engineer? 



Assuming the technology issues were worked out, we still needed to know if students 
and teachers would find talking to the engineer on-line worth their time and effort. 
"Ask the expert” Web sites other than KDN are based on e-mail communication 
where children have plenty of time to formulate questions which are reviewed by 
teachers or parents who correct grammar and edit to ensure clarity. Would children 
be able to communicate well enough to gain usable advice about their gadgets? 
Finding out if students thought talking to the engineer helped them meet the 
challenge was a major focus of the first two field studies and a lesser focus in 
following studies. 

Because children’s access to KDN, and especially the engineer, depended on the 
cooperation of teachers and parents, the grownups’ perception of the value of talking 
to the engineer was important to assess. Would teachers see enough value that they 
would be willing to negotiate the potential maze of scheduling computer time and 
learning to use the Web site? 

What, exactly, should the content of the Web site be? 
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The initial Web site concept included the challenge presented in short story format, a 
printable student journal, a student-engineer chatboard with only text chat, and an on- 
line bulletin board where pictures of gadgets could be posted. The inadequacy of 
these features became apparent very early in development. Identifying what was 
necessary for effective student-engineer communication was a focus early in the 
project. Programming and testing the solutions ultimately took a year longer and 
$70,000 more than was originally planned. 

Phase 1: Field Studies 

Evaluation techniques originally planned (based on the actual grant application)were: 

• Review of materials by DCM staff; 

• Pilot tests (field studies) with small groups of students in schools the first year; 

• Field studies in full classrooms the second year; 

• Written questionnaires and in-depth interviews for teachers with students 
participating in the pilot tests; 

• Observation of pilot studies by DCM staff and the independent evaluator, and 
discussion with participants; 

• Written parent surveys; 

• In-depth case studies of selected students; 

• Collection and evaluation of student materials, including journals, drawings, 
and video, to asses changes in the students' creative problem solving skills, 
ability to transfer knowledge to new situations, and ability to work as teams; 

• Presentation of ongoing evaluation results to the Advisory Board and 
consultation with the members. 

During the first year of the project, two small field studies were conducted at schools 
with which DCM had a strong relationship. The purpose was to assess students' 
reaction to the concept of an on-line engineer, the necessary components for 
effective communication, the appropriate age for KDN, and the appropriateness of the 
first challenge. Teachers chose a few students to participate in KDN. The museum 
brought in their own computers and set up a closed link between the engineer's 
computer and students' computer. Communication was via a FileMakerPro database 
made to resemble a Web page with two-way text chat in the first field study. The 
second study utilized a CU-SeeMe program to provide video chat with an engineer 
via a local area network. 

Much was learned about evaluation during these studies. Written surveys, so 
frequently used in evaluation, yielded minimal valuable information. Parents and 
teachers did not take time to record the detailed observations that are needed for 
comprehensive assessment. The value was also limited because the people 
completing the surveys (parents and teachers) were not present when children were 
working with KDN. Direct observation of students and conversations with them 
yielded the most information. Student journals were collected and an extensive video 
record of the sessions was made. Video and journals proved less valuable for 
evaluation purposes, but very valuable for bringing a new project manager up to 
speed a few months later. 

A third small group trial aimed predominantly at testing a new challenge was held at 
the museum. No engineer was involved in this trial. A Girl Scout leader was asked to 
gather a group of students for this test. Children were asked to read two versions of 
the Puppet Challenge aimed at different age groups. The children were asked 
questions to see if they understood the story and its vocabulary, and if they found it 
interesting. The team wanted to assess the story format, the particular versions of this 
story, whether the challenge provided an appropriately open-ended engineering 
problem solving experience, and to revisit the question of the appropriate age for 
doing KDN. The session was videotaped and the children's journals were collected. 
The independent evaluator was not present. Once again, personal conversations with 
and observations of the children provided the most useful data. 

These very preliminary field studies answered some basic questions and posed new 
ones: 
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• Kids liked talking to the engineer. They found the conversation helpful. 

• Kids in 1 st and 2 nd grade lacked the skills necessary to communicate solely 
through typing. Third graders were marginal. The search for a visual and 
possible audio component to communication was on. 

• It was going to take some fancy programming to pull this off on the Web. 

• The way the challenge was presented needed work in order to make it clear 
and understandable. At that time, challenges were presented in a short story 
format involving a girl and boy and their teacher, Ms. Curious. 

Results of the field studies were presented to the Advisory Board. They felt that the 
potential impact of KDN on science and engineering education was huge, and that 
there were still a lot of questions to be answered. Extensive discussions occurred 
about the best form of communication best (email, video, live text chat, voice, fax, or 
phone), the focus of KDN (science, computers, or engineering), and the importance 
of addressing teachers’ needs. 

A big meeting was held that included the newly hired Web site designers (Eduweb), 
the independent evaluator, the old and new project managers, and most of the DCM 
staff who had worked on KDN to date. A free flowing exchange of ideas made sure 
everyone was on the same page regarding where the project had been and what 
needed to happen in the near future. Eduweb presented information of the most 
appropriate technology available. An understanding of two overarching types of 
problems emerged from the meeting: technology problems and people problems. 

Eduweb focused on finding technology that could provide the best communication, 
could tunnel through school firewalls, and was compatible with the typical aging 
computer systems common in elementary schools. Making the site user-friendly - in 
both graphic design and navigation - and compatible with school computers also 
became a focus. DCM would review everything and try out all interactive components 
of the site. 

DCM focused on the people problems. Most of the team had already concluded that 
1 st and 2 nd graders were too young to use KDN effectively. However, the Museum 
felt that additional data was needed to support this conclusion for the Illinois State 
Board of Education (the funder). It was important to make the teacher, parent and 
child user interested in and comfortable with KDN. This would be addressed through 
both the content of the Web site, and Eduweb’s work on the graphic design and 
navigation. Finding and training on-line engineers continued to be of concern. DCM 
was also responsible for coordinating with schools which had volunteered to 
participate in the second round of field studies. 

Over the next few months the first version of the KDN Web site was developed in 
preparation for the 1 st full-fledged classroom tests. During this period, the DCM 
project manager served as the internal evaluator. Eduweb posted pages on the Web 
for the project manager to review. She typically reviewed the first few iterations of a 
Web page or design element, and then asked for opinions from other DCM staff. 
Preliminary assessment of firewall tunneling technology was tested on a few local 
school computers before being incorporated into the site. Any DCM staff children 
visiting the museum were pressed into service as voluntary evaluators. They tried out 
the functionality of the Engineer chatboard and drawing program, critiqued the 
graphics, and were usually the first to try their hands at a new challenge. 

Because the site was to be accessed by entire classes of children at once, DCM staff 
participated in group tests of the drawing program and the Engineer chatboard. We 
found again and again that even if it works on the programmer’s computer and even 
the project manager’s computer, that does not mean that 10 or 20 or 30 computers 
could draw or use the chatboard at once. The programmer would access error logs 
and make adjustments, and we’d try again another day. 

Phase 2: Classroom Tests 

Scheduling classroom tests proved to be complicated; school calendars and 
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schedules are generally very rigid. An extraordinary amount of accommodation was 
necessary on the part of the school. The whole school was often affected because 
computer lab time and frequently library time were adjusted so that the classes using 
KDN had access to the computers. 

What we did right in these trials 

Administrators at the participating schools were on the KDN Advisory board, so they 
were familiar with the concept behind KDN. These schools and administrators had, 
for the most part, been selected because DCM conducted many programs there and 
had a good relationship with the teachers and administrators. 

Students from 1 st through 4 th grade were involved in this phase of testing, allowing 
collection of data on the age issue. Also important were the responses of teachers 
and students of different ages to the challenges and their presentation, the KDN 
student journals, and students’ desire, ability, and willingness to talk to the engineer. 

To address 1 st and 2 nd graders’ communication difficulties, a drawing program and 
whiteboard with chat component (Engineer chatboard) had been added to the site to 
provide a visual component. Children and engineers could both see the child’s design 
and mark on it. Cell phones were used with and without the whiteboard to talk to the 
engineer as the children were building. 

Three very different schools were chosen: a suburban school in a middleclass 
neighborhood (School 1), a rural, smalltown school (School 2), and a suburban 
school with a large Spanish-speaking population in a low income neighborhood 
(School 3). 

DCM provided a Spanish-speaking facilitator for the bilingual classes at School 3. 

This allowed us to ensure students understood KDN as well as allowing DCM to 
receive feedback from the students. 

Although facilitation was by experienced DCM personnel, classroom teachers were 
present for all KDN sessions and actively participated in the KDN experience. DCM 
and the independent evaluator chatted with teachers before and during KDN sessions 
to find out what teachers wanted and needed before using KDN. Informal 
conversations allowed us to see how revolutionary open-ended, child-centered 
problem solving is to some teachers and what type of background information on the 
philosophy behind KDN would be needed on the Web site. For example, one 2 nd 
grader making a bell puppet asked a DCM facilitator for help early in the process, and 
was later helped by her teacher. At the end of the session, the DCM facilitator asked 
her how her bell turned out. The girl said, ”lt’s not a bell anymore. My teacher 
changed it." It was clear the Web site needed specific guidance on how teachers 
could help children solve their problems instead of taking over their project. 

Eduweb’s project manager and the computer programmer were present at the first 
school where testing occurred. Although both are experienced specialists in children’s 
Web sites, they agreed there was no substitute for seeing children actually trying to 
use the site en masse in the school. It gave them a new appreciation for the state of 
technology in the elementary school, and of the problems of Web site functionality 
with young children. 

Eduweb's computer programmer and project manager both played the role of 
engineer several times during this phase of testing. This gave them first hand 
knowledge of the difficulties of using the engineer’s communication tools. The 
programmer reported that this experience was extremely valuable. For example, 
being limited to only the student whiteboard chat for communication, the engineer 
didn’t know what was happening in the classroom during a long period of inactivity. 
Had the students all left for lunch? The solution was to add a separate chat 
component enabling a private channel of communication with the teacher. 

Major revisions to the drawing program and the Engineer chatboard were made after 
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each school participated. 



The independent evaluator was present for most of the session at all three schools. 
She carefully observed all aspects of the sessions, chatted with students and 
teachers during the sessions, and asked follow-up questions at the end. For those 
sessions not attended, the project manager provided detailed notes and comments. 
Following the evaluator's model, the project manager asked students and teachers 
follow-up questions at the end of their KDN experience. 

After School 1 tests, the team completely revised the challenge format to make it 
clearer. After in-house evaluation at DCM, the new format was tested at School 2, 
revised slightly based on teacher comments, and tested again at School 3. This 
format is still used today. 

What we did wrong in these trials 

The schools had been chosen because their administrators were on the KDN 
Advisory Board, but without regard for the state of their computer systems. A few 
weeks before the first tests, we found out School 1 had very old computers that 
simply could not handle even the most basic aspects of the KDN site. Eventually, the 
district computer expert borrowed a portable iBook lab from the junior high and set it 
up at the elementary school for the KDN tests. Consequently, the children were 
learning to use a new computer and the KDN Web site at the same time. 

Because the administrators, rather than the teachers, had made the decision to 
participate in KDN development, many teachers were less than enthusiastic. Many, 
but not all, were not adept at computer use and did not see the value of hands-on 
activities. However, one bilingual teacher at School 3 enthusiastically reported that 
KDN had sparked a whole class discussion on how "this concept of keep trying until 
you get it right" applies to the rest of life, too. He said that the open-ended problem 
solving of KDN helps teachers see a new way of doing things, too. 

School 3 had no computer lab at all until two months before the KDN trial. Most of 
these children had no home computers and consequently very little computer 
expertise. This group of students, however, proved to be very excited about KDN and 
vividly illustrated the potential impact of KDN on traditionally underserved 
populations. The students thrived when building their gadgets because there was no 
language issue with the hands-on activity. The students easily learned to use the 
Web site's interactive features. They were thrilled to talk to the engineer, and the girls 
especially were excited to learn that the engineer was a woman. However, many of 
their teachers struggled with the computer. These teachers, many of whom had never 
used computers in their classrooms, illustrated the importance of thorough, clear 
guidance in using KDN and user-friendly navigation. 

Because of individual school scheduling constraints, the first trials were with 1 st and 
2 nd graders, the group we expected to have the most trouble with KDN in general. A 
more refined version of the Web site would have given us a truer picture of the 
children's abilities without the complications of computer bugs. However, the tests still 
demonstrated conclusively that these young children were not developmental^ ready 
for KDN. 

Follow-up questionnaires were given to the teachers. Due to the poor return rate on 
written teacher questionnaires, the evaluator scheduled group de-briefing sessions. 
This was a more efficient use of teachers' time, and the group dynamic yielded more 
useful information. For example, a teacher would raise an issue, and several others 
would agree. This helped us see which issues resonated most with teachers. These 
discussions provided valuable, specific data that was used by the project manager 
and programmers to make major changes to the Web site and program content. 

Web Site Usability Tests 

After the classroom tests at the third school, the KDN team met and concluded we 





5/27/2003 



basically had a workable Website, albeit still with many bugs. A primary concern was 
that the site had become so big that navigation was a mess. The home page was 
overloaded with information, and there was a general lack of user-friendliness to the 
site, largely due to its complexity. Because the team was so accustomed to using the 
site, a new pair of eyes was needed for assessment. Two new evaluation strategies 
were adopted: teacher usability studies and review by independent Web site 
developers. 

Teacher Usability Studies 

To identify gaps in information and get teachers’ perspectives, two elementary 
classroom teachers were brought in to look at the Web site in two usability studies. 

We gave the teachers very little orientation to the site because we wanted to observe 
them the first time they saw the Web site. Using a "think-aloud” protocol, the 
evaluator, project manager and Web developer took careful notes as each teacher 
navigated the site while talking aloud. This gave us specific information on where 
navigation and information needed revising. Following their time exploring the site, we 
held debriefing sessions during which teachers explained and expanded upon some 
of their comments. These usability studies were among the most useful techniques 
used in the entire evaluation. 

After the first teacher reviewed the site, major revisions were made. Then a second 
teacher was observed using the same ’'think-aloud" protocol. After a morning of 
learning about the site, we observed as the teacher directed six children, who were 
unfamiliar with KDN, in using the site, including the drawing program and talking to 
the engineer. As the focus of this exercise was on the Web site itself, the children did 
not construct their gadgets. One of the most striking observations was that, prior to 
observing the children, the teacher did not think they would be interested in talking to 
the engineer. She stated she was very surprised to see how enthusiastic they were 
when talking to the engineer, and this changed her opinion of KDN considerably. 

Both teachers pointed up a paradox the KDN team had long been wrestling with: they 
thought there was too much information on the site, but they wanted more 
information. The KDN team discussed this problem and concluded the solution must 
lie in presenting the information in a more eye-appealing way. Once we had a good 
idea what teachers needed on the site, we needed an uninitiated Web designer's 
opinion on how to give it to them. 

Independent Web site Developer Reviews 

After extensive revisions based on the teacher usability studies, two Web site 
developers unfamiliar with KDN were asked to review the site. We were particularly 
interested in their ideas on the overall look and feel of the site - especially graphic 
design and navigation. Although there were some common themes in the reviewers’ 
comments - especially the need to simplify the navigation and put less information on 
each page - the reviewers came up with significantly different solutions to these 
problems. The KDN team evaluated the reviewers' comments and used them as a 
starting point for major revisions to the site navigation, page layout, and site 
organization. 

Analysis and Redesign 

Much discussion occurred between Eduweb and DCM, and many prototypes were 
reviewed before a basic design evolved. We were, in effect, reinventing the wheel 
with four new visions on how to go about it - DCM's, Eduweb's, and the two 
reviewers'. Text on every page was revised, a new graphics shell was developed, site 
content was reorganized and an entirely new navigation scheme developed, colors 
were constantly changed, a new log-in sequence was developed, and major revisions 
- both functional and cosmetic - to the drawing program and engineer chatboard were 
implemented. Revising the KDN home page was a major effort. The home page used 
in Phase 2 classroom trials is shown in Figure 2, and the current home page is shown 
in Figure 3. There were at least a dozen interim versions. The resulting current 
version of the site bears little resemblance to the reviewers' suggestions or the 
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version they reviewed, but based on feedback from teachers who have since used 
the site, most of the problems have been resolved. 




What is KDN? 

pfobSrm #nd sfcHt ch~d^r In nm Uwougft fcwrth gr^dt*. 
OhOdten ctev^i\ buiW and toil * tfcM sob** the ttv 3 dl«r> 3 e~-dn<l 

CArt witfi «n rw; fO* lead-time fa£j>l 



Nave you «v«r wanted to: 

• S^# 0 #d;tardroi«r«w»t 9 r? 

• DdUMaroboetodtan taur room? 

• lnw««tlQit* *'*r* to **rv3 ptopl* to Mart? 

• lnutrrt*oadd*tCh 4 tl<toTourdoglaurxn*ro<t:*tl 

. tfa prablwn ol ho* to kttp t<u' eae »w*t trom your 

Thcte arc all engineering chaiengeri 

D KON fM Mill 0 *t 10 ! 



J. fmj out abaut »♦*! B» > n fc m<fc | 

2 . EixSor* a U/j m#»W >»9 du ll e r, 
J, By»ih a <M4g«t to meet ihe c»> >■»*»♦, 
4 . Sutd 



tfa KDW W*b Ute 4 
In dartrioement. Veen 
bod many teebon* 
ftrf wdir 

eemoueber*. Plea*# 
fee* fat t» expire 
the utt **J"t uvt r 

rrvrvd , 



Leant shchc about 
ON fcf lL a ibt£I «r 
&f pittsck (teutnj 
wn> 



U>fh 

L*»out 



<MMrr 3 t* 
1 Orrz* 



Jp 



ym! in* the gmuuU* to: 

« Of aw ^ 4 etle*fc* o**?*t, 

• Tan: to an rni^r 

• See *hat Otr«r IM w|bt m fart detlontd and Mil to meet toe challenge . 



/"mart now 



At 1 MftM YOU Are The Engineer! 



om %^-t 

«ab qX X o 



•eoxrt on m* 




•UKdyM GLAMfTt 
tjAitoW* > 
Cm Inanttf 

* |H M l 







out Se* 



Ml D* >>v* "**>«** Ir b*«*e 4 *▼■» lo I>f <3 **t>i t-v*n 

l*rti t4«H e# % o«nt>e L**'*ty t'trt (ri'-A 

u>* w*b it* a*iu^i*a *fci d*«*tso*a i * ti* jf.it in»i aj&jtirkl&tUi 
Copy^/n Ml Ou(>*o« CMduft'i t4u»*-m. *it *fta4 nixvid 
hnptff***. A»e»©*eU<U*f trww • uflkiig 



mow 

CuPa,,* 

<Mdren'* 

Museum 



Figure 2: KDN home page used in Phase 2 classroom trials . 



This page was too long, too busy and generally notuser-friendly. 
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The solution to too much information was the creation of a KDN "Quick Tour" that 
takes visitors through a virtual tour explaining how KDN works; and a "What is KDN?" 
section providing additional details. 

Phase 3 Classroom Trials 

The plan for Phase 3 Classroom Trials was for classroom teachers to facilitate KDN 
with only e-mail and phone support from DCM. Two teachers, one in a small town 
near Chicago and one in an inner city Chicago school, found KDN on the Web on 
their own. They provided the first real test of the extensive information in the 
Teacher's Guide section of the Web site, and the newly revised Web site. The 
schools' proximity to DCM allowed the KDN Project Manager and, in one case, the 
independent evaluator to visit the schools and observe as students built their gadgets 
and talked to the engineer. The evaluator and project manager conducted an 
extensive interview with the teachers. Their feedback was used to further refine the 
Web site. 

We were relieved to see that these children's experience was remarkably similar to 
KDN activities facilitated by DCM staff. The teachers were able to glean the 
philosophy behind KDN from the Website, navigate the site, and use its information to 
produce a hands-on, minds-on learning activity for their classes. Each teacher had 
added an individual twist to the challenge to emphasize different concepts, but the 
child-centered, creativity and problem solving concepts were paramount. The 
teachers had successfully used KDN exactly as was intended. 

However, there were still problems with the engineer chatboard, most notably 
stemming from a modification that allowed the engineer to simultaneously chat with 
several students. Ultimately, this feature was disabled because it did not function 
reliably. 

Conclusion 

Although the basic concept behind KDN has remained constant, KDN as originally 
envisioned by DCM was very different from today's version. From the early stages of 
development, the team placed emphasis on continuous evaluation and a willingness 
to make revisions. The collaboration in evaluation between professional evaluator, 
Web site developers, museum personnel and engineers resulted in a fluid 
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development process that mimicked the engineering problem solving process KDN is 
designed to teach. This iterative nature of engineering design as applied to KDN 
development ultimately produced a user-friendly experience consistent with the 
Museum's mission. Our efforts were rewarded with a Gold Muse for Programs 
Emphasizing Two-way Communication from our peers at the American Association of 
Museums. However, the best rewards come from the children who say "WOW! This 
is so cool! I wanna be an engineer when I grow up!" 
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